Skip to content

Fix: last completion chunk with finish_reason not sent in edge case scenario #515

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Nov 5, 2025
Merged

Fix: last completion chunk with finish_reason not sent in edge case scenario #515

merged 4 commits into from
Nov 5, 2025

Conversation

@declark1
Copy link
Collaborator

@declark1 declark1 commented Nov 3, 2025 •
edited
Loading

An edge case was discovered where the last completion chunk containing finish_reason may not be sent with the final batch if it has empty choice text. This is because chunks without choice text are not sent to the detection pipeline.

This PR adds logic to track chunk indices for each batch and send the last completion chunk if not already sent with the final batch.

Closes #516

Example

Request

{
	"model": "ibm-granite/granite-3.0-8b-instruct",
	"prompt": "Please repeat the words in [], do not trim space.\n[I love this world.]",
	"stream": true,
	"seed": 1337,
	"top_p": 1,
	"include_stop_str_in_output": true,
	"repetition_penalty": 1,
	"stream_options": {
		"include_usage": true,
		"continuous_usage_stats": true
	},
	"detectors": {
		"output": {
			"hap": {}
		}
	}
}

Response (before fix)

data: {"id":"cmpl-146db2f386ec4cd2bec44e657604c9eb","object":"text_completion","created":1762209902,"model":"ibm-granite/granite-3.0-8b-instruct","choices":[{"index":0,"text":"\nI love this world.","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":{"prompt_tokens":18,"total_tokens":24,"completion_tokens":6},"detections":{"output":[{"choice_index":0,"results":[]}]}}

data: {"id":"cmpl-146db2f386ec4cd2bec44e657604c9eb","object":"text_completion","created":1762209902,"model":"ibm-granite/granite-3.0-8b-instruct","choices":[],"usage":{"prompt_tokens":18,"total_tokens":25,"completion_tokens":7}}

data: [DONE]

Response (after fix)

data: {"id":"cmpl-146db2f386ec4cd2bec44e657604c9eb","object":"text_completion","created":1762209902,"model":"ibm-granite/granite-3.0-8b-instruct","choices":[{"index":0,"text":"\nI love this world.","logprobs":null,"finish_reason":null,"stop_reason":null}],"usage":{"prompt_tokens":18,"total_tokens":24,"completion_tokens":6},"detections":{"output":[{"choice_index":0,"results":[]}]}}

data: {"id":"cmpl-146db2f386ec4cd2bec44e657604c9eb","object":"text_completion","created":1762209902,"model":"ibm-granite/granite-3.0-8b-instruct","choices":[{"index":0,"text":"","logprobs":null,"finish_reason":"stop","stop_reason":null}],"usage":{"prompt_tokens":18,"total_tokens":25,"completion_tokens":7}}

data: {"id":"cmpl-146db2f386ec4cd2bec44e657604c9eb","object":"text_completion","created":1762209902,"model":"ibm-granite/granite-3.0-8b-instruct","choices":[],"usage":{"prompt_tokens":18,"total_tokens":25,"completion_tokens":7}}

data: [DONE]

@declark1 declark1 force-pushed the fix-finish-reason-edge-case branch from 6b7370e to 92097d8 Compare November 3, 2025 22:56
@declark1 declark1 force-pushed the fix-finish-reason-edge-case branch from 85c5158 to a852ac4 Compare November 3, 2025 23:07
@declark1 declark1 changed the title Fix: completions finish_reason edge case Fix: last completion chunk with finish_reason not sent in edge case scenario Nov 4, 2025
@declark1
Update comments
1801ec5 
Signed-off-by: declark1 <[email protected]>
@declark1 declark1 force-pushed the fix-finish-reason-edge-case branch from 007e4a4 to 1801ec5 Compare November 4, 2025 16:45
@declark1 declark1 marked this pull request as ready for review November 4, 2025 19:43
evaline-ju
evaline-ju approved these changes Nov 4, 2025
View reviewed changes
Copy link
Collaborator

@evaline-ju evaline-ju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@declark1 declark1 merged commit f7fc558 into foundation-model-stack:main Nov 5, 2025
3 checks passed
@declark1 declark1 deleted the fix-finish-reason-edge-case branch November 5, 2025 16:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@evaline-ju evaline-ju evaline-ju approved these changes

@gkumbhat gkumbhat Awaiting requested review from gkumbhat gkumbhat is a code owner

@mdevino mdevino Awaiting requested review from mdevino mdevino is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Last completion chunk with finish_reason not sent in edge case scenario

2 participants

@declark1 @evaline-ju